{
 "cells": [
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": [
    "\n",
    "## 1、暂退法概述\n",
    "神经网络中的“暂退法”通常指的是“Dropout”技术。Dropout 是一种非常有效的正则化方法，用于减少神经网络的过拟合现象。在训练过程中，通过随机地关闭（即设置为零）一部分神经元的输出来实现，这样可以防止模型对特定输入数据过于敏感，从而提高模型的泛化能力。\n",
    "\n",
    "### Dropout 的工作原理\n",
    "\n",
    "1. **随机失活**：在每次前向传播时，根据预设的概率 p（通常是 0.5），随机选择一些神经元进行失活。这意味着这些神经元及其连接不会参与到当前的前向传播和反向传播中。\n",
    "\n",
    "2. **权重缩放**：为了避免因为部分神经元被关闭而导致输出值的大幅度变化，在测试阶段会将所有权重乘以一个因子（通常是 1-p）。这一步骤确保了训练和测试时网络的行为保持一致。\n",
    "\n",
    "### 优点\n",
    "\n",
    "- **减少过拟合**：通过使模型学习到更鲁棒的特征表示，而不是依赖于某些特定的数据特征。\n",
    "- **简化模型**：虽然实际上并没有减少模型参数的数量，但是通过随机关闭神经元的方式减少了模型的有效复杂度。\n",
    "\n",
    "### 应用场景\n",
    "\n",
    "Dropout 技术广泛应用于各种深度学习模型中，尤其是在图像识别、自然语言处理等领域。它对于大型且复杂的模型尤其有效，能够显著提升模型的性能。\n",
    "\n",
    "### 注意事项\n",
    "\n",
    "尽管 Dropout 能够有效减少过拟合，但并不意味着应该在所有情况下都使用它。例如，在模型本身已经相对简单或数据量非常大的情况下，可能没有必要引入 Dropout。此外，设置合适的 dropout 概率也很重要，过高或过低的概率都可能导致模型性能下降。\n",
    "\n",
    "总之，Dropout 是一种强大的工具，合理使用可以大大提升神经网络模型的表现。"
   ],
   "id": "5deb1e6decf0fe89"
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 2、导包",
   "id": "7bc43763162d6b3c"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:45:51.318131Z",
     "start_time": "2024-11-13T05:45:51.313041Z"
    }
   },
   "cell_type": "code",
   "source": [
    "import torch\n",
    "import torchvision\n",
    "from torchvision import datasets, transforms\n",
    "from torch.utils.data import Dataset, DataLoader"
   ],
   "id": "5306e27d4ea784ca",
   "outputs": [],
   "execution_count": 32
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 3、单层暂退法函数",
   "id": "6402bc4a55988c70"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:45:51.401624Z",
     "start_time": "2024-11-13T05:45:51.395646Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def dropout_layer(X, p):\n",
    "    assert 0 <= p <= 1\n",
    "    # 在本情况中，所有的元素被丢弃\n",
    "    if p == 1:\n",
    "        return torch.zeros_like(X)\n",
    "    # 在本情况中，所有元素被保留\n",
    "    if p == 0:\n",
    "        return X\n",
    "    mask = (torch.rand(X.shape) > p).float()\n",
    "    return X * mask / (1.0 - p)"
   ],
   "id": "6f9e62e67fb447f1",
   "outputs": [],
   "execution_count": 33
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 4、测试",
   "id": "d3b91f59b9863d06"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:45:51.417064Z",
     "start_time": "2024-11-13T05:45:51.410153Z"
    }
   },
   "cell_type": "code",
   "source": [
    "X = torch.arange(0., 16.).float()\n",
    "print(X)\n",
    "print(dropout_layer(X, 0.))\n",
    "print(dropout_layer(X, 0.5))\n",
    "print(dropout_layer(X, 1.0))"
   ],
   "id": "59dc0c87470ed44",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,\n",
      "        14., 15.])\n",
      "tensor([ 0.,  1.,  2.,  3.,  4.,  5.,  6.,  7.,  8.,  9., 10., 11., 12., 13.,\n",
      "        14., 15.])\n",
      "tensor([ 0.,  2.,  4.,  6.,  8.,  0., 12., 14.,  0., 18.,  0.,  0.,  0.,  0.,\n",
      "         0.,  0.])\n",
      "tensor([0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])\n"
     ]
    }
   ],
   "execution_count": 34
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 5、定义模型参数",
   "id": "8ee10d748b5daa3"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:45:51.421534Z",
     "start_time": "2024-11-13T05:45:51.418729Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 输入、输出、第一个隐藏层、第二个隐藏层\n",
    "num_inputs,num_outputs,num_hiddens1,num_hiddens2 = 784,10,256,256"
   ],
   "id": "c770a7d4098ae4ce",
   "outputs": [],
   "execution_count": 35
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 6、定义模型",
   "id": "94ac5b36264aaaa4"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:45:51.432558Z",
     "start_time": "2024-11-13T05:45:51.424829Z"
    }
   },
   "cell_type": "code",
   "source": [
    "dropout1 = 0.2 # 第一层暂退概率\n",
    "dropout2 = 0.5 # 第二层暂退概率\n",
    "\n",
    "class Net(torch.nn.Module):\n",
    "    def __init__(self, num_inputs,num_outputs,num_hiddens1,num_hiddens2,is_training=True):\n",
    "        super(Net, self).__init__()\n",
    "        self.num_inputs = num_inputs\n",
    "        self.num_outputs = num_outputs\n",
    "        self.num_hiddens1 = num_hiddens1\n",
    "        self.num_hiddens2 = num_hiddens2\n",
    "        self.is_training = is_training\n",
    "        self.lin1 = torch.nn.Linear(num_inputs,num_hiddens1)\n",
    "        self.lin2 = torch.nn.Linear(num_hiddens1,num_hiddens2)\n",
    "        self.lin3 = torch.nn.Linear(num_hiddens2,num_outputs)\n",
    "        self.relu = torch.nn.ReLU()\n",
    "    def forward(self, X):\n",
    "        # 对输入层增加激活函数\n",
    "        H1 = self.relu(self.lin1(X.reshape(-1, self.num_inputs)))\n",
    "        # 只在训练时使用暂退法\n",
    "        if self.is_training:\n",
    "            # 在第一个全连接层之后添加一个暂退层\n",
    "            H1 = dropout_layer(H1, dropout1)\n",
    "        H2 = self.relu(self.lin2(H1))\n",
    "        if self.is_training:\n",
    "            # 在第二个全连接层之后添加一个暂退层\n",
    "            H2 = dropout_layer(H2, dropout2)\n",
    "        out = self.lin3(H2)\n",
    "        return out\n",
    "net = Net(num_inputs,num_outputs,num_hiddens1,num_hiddens2, True)    "
   ],
   "id": "789afbf99edfca13",
   "outputs": [],
   "execution_count": 36
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 7、训练和测试",
   "id": "8bfbdf3438a559a9"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:49:26.095990Z",
     "start_time": "2024-11-13T05:45:51.455717Z"
    }
   },
   "cell_type": "code",
   "source": [
    "num_epochs, lr, batch_size = 10, 0.001, 10\n",
    "loss_fn = torch.nn.CrossEntropyLoss(reduction='none')\n",
    "optimizer = torch.optim.SGD(net.parameters(), lr=lr)\n",
    "# 定义数据变换\n",
    "transform = transforms.Compose([\n",
    "    transforms.ToTensor(),  # 将图像转换为 Tensor\n",
    "    transforms.Normalize((0.5,), (0.5,))  # 标准化\n",
    "])\n",
    "\n",
    "# 下载并加载训练数据\n",
    "train_dataset = torchvision.datasets.MNIST(root='../data', train=True, download=True, transform=transform)\n",
    "train_loader = DataLoader(train_dataset, batch_size=batch_size, shuffle=True)\n",
    "\n",
    "# 下载并加载测试数据\n",
    "test_dataset = torchvision.datasets.MNIST(root='../data', train=False, download=True, transform=transform)\n",
    "test_loader = DataLoader(test_dataset, batch_size=batch_size, shuffle=False)\n",
    "\n",
    "for epoch in range(num_epochs):\n",
    "    for X,y in train_loader:\n",
    "        outputs = net(X)\n",
    "        # 梯度清零\n",
    "        optimizer.zero_grad()\n",
    "        # 计算损失率\n",
    "        l = loss_fn(outputs, y)\n",
    "        l.sum().backward() # 反向传播\n",
    "        optimizer.step() # 更新参数\n",
    "        \n"
   ],
   "id": "9110d83f059c9796",
   "outputs": [],
   "execution_count": 37
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:49:26.106427Z",
     "start_time": "2024-11-13T05:49:26.097641Z"
    }
   },
   "cell_type": "code",
   "source": [
    "def get_labels(labels):\n",
    "    text_labels = [\"t-shirt(T恤)\", \"trouser(裤子)\", \"pullover(套衫)\", \"dress(连衣裙)\", \"coat(外套)\", \"sandal(凉鞋)\",\"shirt(衬衫)\", \"sneaker(运动鞋)\",\n",
    "                   \"bag(包)\", \"ankle boot(短靴)\"]\n",
    "    return [text_labels[int(label)] for label in labels ]\n",
    "for X,y in test_loader:\n",
    "    break;\n",
    "trues = get_labels(y)\n",
    "preds = get_labels(net(X).argmax(axis=1))\n",
    "print(trues)\n",
    "print(preds)  "
   ],
   "id": "b66058407c80fc22",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sneaker(运动鞋)', 'pullover(套衫)', 'trouser(裤子)', 't-shirt(T恤)', 'coat(外套)', 'trouser(裤子)', 'coat(外套)', 'ankle boot(短靴)', 'sandal(凉鞋)', 'ankle boot(短靴)']\n",
      "['sneaker(运动鞋)', 'pullover(套衫)', 'trouser(裤子)', 't-shirt(T恤)', 'coat(外套)', 'trouser(裤子)', 'coat(外套)', 'ankle boot(短靴)', 'sandal(凉鞋)', 'ankle boot(短靴)']\n"
     ]
    }
   ],
   "execution_count": 38
  },
  {
   "metadata": {},
   "cell_type": "markdown",
   "source": "## 8、简洁实现",
   "id": "e6ac4a8b92c06104"
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:53:01.462586Z",
     "start_time": "2024-11-13T05:49:26.108756Z"
    }
   },
   "cell_type": "code",
   "source": [
    "# 1、定义模型\n",
    "model = torch.nn.Sequential(\n",
    "    torch.nn.Flatten(),\n",
    "    torch.nn.Linear(num_inputs, num_hiddens1),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Dropout(dropout1),\n",
    "    torch.nn.Linear(num_hiddens1, num_hiddens2),\n",
    "    torch.nn.ReLU(),\n",
    "    torch.nn.Dropout(dropout2),\n",
    "    torch.nn.Linear(num_hiddens2, num_outputs),\n",
    ")\n",
    "# 2、配置参数\n",
    "def init_weights(m):\n",
    "    if type(m) == torch.nn.Linear:\n",
    "        torch.nn.init.normal_(m.weight, std=0.01)\n",
    "model.apply(init_weights)\n",
    "# 3、定义优化器\n",
    "optimizer = torch.optim.SGD(model.parameters(), lr=0.001)\n",
    "# 4、定义损失函数\n",
    "loss_fn = torch.nn.CrossEntropyLoss(reduction='none')\n",
    "# 5、训练\n",
    "for epoch in range(num_epochs):\n",
    "    # model.train()\n",
    "    for X,y in train_loader:\n",
    "        outputs = model(X)\n",
    "        optimizer.zero_grad()\n",
    "        l = loss_fn(outputs, y)\n",
    "        l.mean().backward()\n",
    "        optimizer.step()\n",
    "# 6、预测        \n",
    "def get_labels(labels):\n",
    "    text_labels = [\"t-shirt(T恤)\", \"trouser(裤子)\", \"pullover(套衫)\", \"dress(连衣裙)\", \"coat(外套)\", \"sandal(凉鞋)\",\"shirt(衬衫)\", \"sneaker(运动鞋)\",\n",
    "               \"bag(包)\", \"ankle boot(短靴)\"]\n",
    "    return [text_labels[int(label)] for label in labels ]\n",
    "for X,y in test_loader:\n",
    "    break;\n",
    "trues = get_labels(y)\n",
    "preds = get_labels(model(X).argmax(axis=1))\n",
    "print(trues)\n",
    "print(preds)       "
   ],
   "id": "2871d9cb96f40937",
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "['sneaker(运动鞋)', 'pullover(套衫)', 'trouser(裤子)', 't-shirt(T恤)', 'coat(外套)', 'trouser(裤子)', 'coat(外套)', 'ankle boot(短靴)', 'sandal(凉鞋)', 'ankle boot(短靴)']\n",
      "['sneaker(运动鞋)', 'pullover(套衫)', 'trouser(裤子)', 't-shirt(T恤)', 'coat(外套)', 'trouser(裤子)', 'coat(外套)', 'ankle boot(短靴)', 'shirt(衬衫)', 'ankle boot(短靴)']\n"
     ]
    }
   ],
   "execution_count": 39
  },
  {
   "metadata": {
    "ExecuteTime": {
     "end_time": "2024-11-13T05:53:01.466744Z",
     "start_time": "2024-11-13T05:53:01.464399Z"
    }
   },
   "cell_type": "code",
   "source": "",
   "id": "c1454aaa325ce5c5",
   "outputs": [],
   "execution_count": 39
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
